Neural Network Training Research Articles

Approximate Dynamic Programming for Control of Wave Energy Converters With Implementation and Validation on a Point Absorber Prototype

It is recognized that the application of constrained optimal control for wave energy converters (WECs), represented by model predictive control (MPC), is hindered by its computation burden resulting from online optimization, especially when the model contains nonlinearities. In this article, a novel control solution based on approximate dynamic programming (ADP) is proposed, in which an explicit controller is solved by model sampling and neural network training. In this way, the major computation load is moved offline to enable fast online execution, and the need for wave prediction is eliminated. The effectiveness of ADP is verified by a simulation case study based on a nonlinear WEC system. Using nonlinear MPC as the performance benchmark, over 90% energy efficiency is achieved by ADP under various sea states with constraint satisfaction and dramatically faster computation. The ADP controller is further experimentally validated on a newly constructed WEC prototype through wave tank testing, where successful implementation and satisfactory performance are demonstrated.

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

Brave the Wind and the Waves: Discovering Robust and Generalizable Graph Lottery Tickets.

The training and inference of Graph Neural Networks (GNNs) are costly when scaling up to large-scale graphs. Graph Lottery Ticket (GLT) has presented the first attempt to accelerate GNN inference on large-scale graphs by jointly pruning the graph structure and the model weights. Though promising, GLT encounters robustness and generalization issues when deployed in real-world scenarios, which are also long-standing and critical problems in deep learning ideology. In real-world scenarios, the distribution of unseen test data is typically diverse. We attribute the failures on out-of-distribution (OOD) data to the incapability of discerning causal patterns, which remain stable amidst distribution shifts. In traditional spase graph learning, the model performance deteriorates dramatically as the graph/network sparsity exceeds a certain high level. Worse still, the pruned GNNs are hard to generalize to unseen graph data due to limited training set at hand. To tackle these issues, we propose the Resilient Graph Lottery Ticket (RGLT) to find more robust and generalizable GLT in GNNs. Concretely, we reactivate a fraction of weights/edges by instantaneous gradient information at each pruning point. After sufficient pruning, we conduct environmental interventions to extrapolate potential test distribution. Finally, we perform last several rounds of model averages to further improve generalization. We provide multiple examples and theoretical analyses that underpin the universality and reliability of our proposal. Further, RGLT has been experimentally verified across various independent identically distributed (IID) and out-of-distribution (OOD) graph benchmarks.

Random feedback alignment algorithms to train neural networks: Why do they align?

Abstract Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. The performance of FA is generally attributed to an alignment of the update of the random walker with the true gradient --- the eponymous gradient alignment --- which drives an approximate gradient descent. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descent.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

Deep Learning Analysis with Gray Scale and Doppler Ultrasonography Images to Differentiate Graves' Disease.

Thyrotoxicosis requires accurate and expeditious differentiation between Graves' disease (GD) and thyroiditis to ensure effective treatment decisions. This study aimed to develop a machine learning algorithm using ultrasonography and Doppler images to differentiate thyrotoxicosis subtypes, with a focus on GD. This study included patients who initially presented with thyrotoxicosis and underwent thyroid ultrasonography at a single tertiary hospital. A total of 7,719 ultrasonography images from 351 patients with GD and 2,980 images from 136 patients with thyroiditis were used. Data augmentation techniques were applied to enhance the algorithm's performance. Two deep learning models, Xception and EfficientNetB0_2, were employed. Performance metrics such as accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score were calculated for both models. Image pre-processing, neural network model generation, and neural network training results verification were performed using DEEP:PHI® platform. The Xception model achieved 84.94% accuracy, 89.26% sensitivity, 73.17% specificity, 90.06% PPV, 71.43% NPV, and an F1 score of 89.66 for the diagnosis of GD. The EfficientNetB0_2 model exhibited 85.31% accuracy, 90.28% sensitivity, 71.78% specificity, 89.71% PPV, 73.05% NPV, and an F1 score of 89.99. Machine learning models based on ultrasound and Doppler images showed promising results with high accuracy and sensitivity in differentiating GD from thyroiditis.

A Novel Hybrid Approach to the Diagnosis of Simultaneous Imbalance and Shaft Bowing Faults in a Jeffcott Rotor-Bearing System

Ensuring optimal performance and reliability in rotor-bearing systems is crucial for industrial applications. Imbalances and shaft bowing in these systems can lead to decreased efficiency and increased vibrations. The early detection and mitigation of a rotor’s faults are essential, and model-based fault identification has gained much attention in the manufacturing industry over the years. Over the past two decades, however, the development of fault diagnosis rules with data-driven and artificial intelligence (AI) methods has become a trend, and in the foreseeable future the combination of AI with big data will become mainstream. Nevertheless, the critical role of rotating machinery in manufacturing introduces a challenge, as often insufficient fault data are available. This limitation renders the establishment of diagnostic rules using data-driven methods and AI technologies impractical. In light of these challenges, this study proposes a novel hybrid approach that combines a physical model with machine learning (ML) techniques for the diagnosis of multi-faults (imbalances and shaft bowing are demonstrated) in a Jeffcott rotor. To overcome the lack of real-world labeled fault datasets, a physics-based Jeffcott rotor model is first derived and then used to generate abundant fault datasets for ML. Subsequently, simulated data are employed for the training of an artificial neural network (ANN), enabling the network to learn from and analyze the vast array of generated data. The results prove that a well-trained feed-forward neural network (FNN) can accurately isolate and diagnose imbalance and shaft bowing faults using the simulated and real data from the Jeffcott rotor experiment. These physics-based and ML approaches prove effective particularly for multi-faults, offering new possibilities for advanced rotor system monitoring and maintenance strategies in industrial applications.

WiKA: A Vision Based Sign Language Recognition from Extracted Hand Joint Features using DeepLabCut

The use of hand gesture recognition for sign language translation to address the communication gap between the hearing majority and the deaf population has had significant breakthroughs over the years. While the contact-based approach uses wearable devices, a vision-based solution is preferred owing to the convenience it offers and since it obviates the need for complicated gears. This study presents the development of WiKA, an open-source software designed to track the joints of the hands and interpret them into their corresponding sign language counterparts. DeepLabCut, a markerless pose estimation software, was employed to develop the Hand-Joint Tracking Model through the training of a sequential Convolutional Neural Network, utilizing extracted Hand-Joint features to predict the sign language alphabets (A-Z) and numbers (1-9) based on the positioning of the joints. The developed Hand-Joint tracking model exhibited a 4.92% training error and a 5.74% test error with a p-cut-off of 60%. On the other hand, the developed sign language recognition achieved a 96.44% prediction accuracy with only 0.0356% misclass. This model can be further integrated into mobile phones for seamless conversations between the signing and non-signing populations.

Enhancing Training of Physics-Informed Neural Networks Using Domain Decomposition–Based Preconditioning Strategies

Enhancing Training of Physics-Informed Neural Networks Using Domain Decomposition–Based Preconditioning Strategies

Physics-Informed Neural Networks for High-Frequency and Multi-Scale Problems Using Transfer Learning

Physics-Informed Neural Network (PINN) is a data-driven solver for partial and ordinary differential equations (ODEs/PDEs). It provides a unified framework to address both forward and inverse problems. However, the complexity of the objective function often leads to training failures. This issue is particularly prominent when solving high-frequency and multi-scale problems. We proposed using transfer learning to boost the robustness and convergence of training PINN, starting training from low-frequency problems and gradually approaching high-frequency problems through fine-tuning. Through two case studies, we discovered that transfer learning can effectively train PINNs to approximate solutions from low-frequency problems to high-frequency problems without increasing network parameters. Furthermore, it requires fewer data points and less training time. We compare the PINN results using direct differences and L2 relative error showing the advantage of using transfer learning techniques. We describe our training strategy in detail, including optimizer selection, and suggest guidelines for using transfer learning to train neural networks to solve more complex problems.

Deep-learning-assisted design of multi-degree-of-freedom metamaterial absorber

In recent years, the development of metamaterial absorbers (MAs) based on deep learning methods has become a popular research topic. Nevertheless, conventional analytical approaches have limitations, leading to MAs characterized by low-degree-of-freedom cell structures, thereby restricting their absorption characteristics. In this study, we propose a multi-degree-of-freedom MA with multiple variable parameters in terms of structures, materials, and number of layers, which are distinguished by coding. Additionally, deep learning models, including a fully connected neural network, recurrent neural network, AlexNet, and residual neural network, are designed to predict structures and absorption spectra. We compare the prediction accuracy of these four neural network models and identify the optimal network model for spectral prediction and on-demand design. Using trained neural network models, we successfully designed broadband, dual-band, and single-band MAs in on-demand design. Compared with other design methods, this method provides a greater degree of freedom in device design, which allows the design of MAs to have more absorption characteristics

Analysis of the Effectiveness of Using Two-Stage Neural Network Models for Early Detection of Forest Fires

The purpose of the research – analysis of the effectiveness of two-stage neural network models for solving the problem of detecting forest fires in images obtained from unmanned aerial vehicles.Methods. А training dataset was synthesized for training neural network models for the purpose of detection and semantic segmentation of forest fires in images. Тwo-stage neural network models (“Faster R-CNN”, “Mask RCNN” and “Retina-Net”) were used for training. Тhe neural network models were trained according to the same parameters set for all models in order to ensure consistency and a common basis for experiments. Optimization of model parameters during the training process was carried out to minimize the classification loss function. Тo synthesize the test sample, we used a video sequence covering the events of forest fires in the /rkutsk region, which was filmed by an unmanned aerial vehicle. Using a specially developed script in the Рython programming language, the process of dividing this video sequence into separate frames was carried out, which were used as a test data set when assessing the quality of classification of trained neural network models.Results. Based on the analysis of the obtained values of the quality criterion, as well as visual analysis on the test data set produced as part of testing neural network models, the effectiveness of the studied models for detecting forest fires in images was assessed. Тo assess the quality of binary classification of neural network models, the quality criterion “Accuracy” (classification accuracy) was used.Conclusion. Еxperimental studies on a test data set showed that the Retina-Net model demonstrates the lowest, but acceptable, performance compared to other studied neural network models. Тhe two-stage neural network models “Faster R-CNN” and “Mask R-CNN” demonstrate similar classification accuracy values (0.9492 and 0.9521, respectively), which allows us to recommend them for use in early detection systems for forest fires.

Neural Network Modeling of an SLA Printed Mesostructure

Abstract This paper addresses the scarcity of comprehensive studies on the collective impact of various parametric lattice designs on mesostructure functionality. Focusing on optimizing the energy absorption of a serpentine mesostructure made using SLA, this research leverages a feedforward neural network to explore the interplay between line width, number of turns, and material properties on the energy absorbed by the structure. Compression simulations using a finite element model, covering a range of configurations, provided the dataset for neural network training. The resulting network was used to probe correlations between geometric variables, material, and energy absorption. Additionally, a neural network sensitivity analysis explored the impact of hidden layers and number of neurons on the network's performance, demonstrating the network's robustness. The optimized mesostructure configuration, identified by the neural network, maximized energy absorption. Using foundational mechanics of materials concepts, the discussion explains the how the geometry and material of the cellular mesostructure affects structural stiffness.

Enhancing Microwave Photonic Interrogation Accuracy for Fiber-Optic Temperature Sensors via Artificial Neural Network Integration

In this paper, an application of an artificial neural network algorithm is proposed to enhance the accuracy of temperature measurement using a fiber-optic sensor based on a Fabry–Perot interferometer (FPI). It is assumed that the interrogation of the FPI is carried out using an optical comb generator realizing a microwave photonic approach. Firstly, modelling of the reflection spectrum of a Fabry–Perot interferometer is implemented. Secondly, probing of the obtained spectrum using a comb-generator model is performed. The resulting electrical signal of the photodetector is processed and is used to create a sample for artificial neural network training aimed at temperature detection. It is demonstrated that the artificial neural network implementation can predict temperature variations with an accuracy equal to 0.018 °C in the range from −10 to +10 °C and 0.147 in the range from −15 to +15 °C.

Innovative approach to predict soil moisture using the backpropagation‐Elman neural network

AbstractAccurate moisture prediction aids in taking early preventive measures to minimize damage caused by severe storms, droughts, and floods. The study collected 24 types of statistical data from the Xilingol Grassland, Inner Mongolia, spanning from 2011 to 2021. By integrating Elman, backpropagation (BP), and soil moisture prediction model (Elman‐BP) neural networks, soil moisture to 2 m was predicted. The soil moisture mechanism model analysis revealed that soil moisture is related to water supply rather than directly correlated with time. Based on these findings, a two‐stage soil moisture prediction model was established. In the first stage, the Elman neural network model predicted future precipitation data based on observed values. In the second stage, soil moisture was predicted using a BP neural network model based on predicted precipitation data. Comparing the mean squared errors of training and prediction data across the three models revealed that the Elman‐BP neural network had superior predictive accuracy. The maximum relative errors for the trained Elman neural network, BP neural network, and Elman‐BP model were 31.35%, 20.98%, and 3.94%. Overall, results showed a link between soil moisture and water supply and that soil moisture decreased with increasing temperatures. Soil moisture at 10‐ and 40‐cm depths was increased with increasing rainfall. Furthermore, a delay exists between the 40‐ and 100‐cm soil depths. Moreover, soil moisture stability increased with depth.

ESF-YOLO: an accurate and universal object detector based on neural networks

As an excellent single-stage object detector based on neural networks, YOLOv5 has found extensive applications in the industrial domain; however, it still exhibits certain design limitations. To address these issues, this paper proposes Efficient Scale Fusion YOLO (ESF-YOLO). Firstly, the Multi-Sampling Conv Module (MSCM) is designed, which enhances the backbone network’s learning capability for low-level features through multi-scale receptive fields and cross-scale feature fusion. Secondly, to tackle occlusion issues, a new Block-wise Channel Attention Module (BCAM) is designed, assigning greater weights to channels corresponding to critical information. Next, a lightweight Decoupled Head (LD-Head) is devised. Additionally, the loss function is redesigned to address asynchrony between labels and confidences, alleviating the imbalance between positive and negative samples during the neural network training. Finally, an adaptive scale factor for Intersection over Union (IoU) calculation is innovatively proposed, adjusting bounding box sizes adaptively to accommodate targets of different sizes in the dataset. Experimental results on the SODA10M and CBIA8K datasets demonstrate that ESF-YOLO increases Average Precision at 0.50 IoU (AP50) by 3.93 and 2.24%, Average Precision at 0.75 IoU (AP75) by 4.77 and 4.85%, and mean Average Precision (mAP) by 4 and 5.39%, respectively, validating the model’s broad applicability.

Using convolutional neural networks for stereological characterization of 3D hetero-aggregates based on synthetic STEM data

The 3D nano/microstructure of materials can significantly influence their macroscopic properties. In order to enable a better understanding of such structure-property relationships, 3D microscopy techniques can be deployed, which are however often expensive in both time and costs. Often 2D imaging techniques are more accessible, yet they have the disadvantage that the 3D nano/microstructure of materials cannot be directly retrieved from such measurements. The motivation of this work is to overcome the issues of characterizing 3D structures from 2D measurements for hetero-aggregate materials. For this purpose, a method is presented that relies on machine learning combined with methods of spatial stochastic modeling for characterizing the 3D nano/microstructure of materials from 2D data. More precisely, a stochastic model is utilized for the generation of synthetic training data. This kind of training data has the advantage that time-consuming experiments for the synthesis of differently structured materials followed by their 3D imaging can be avoided. More precisely, a parametric stochastic 3D model is presented, from which a wide spectrum of virtual hetero-aggregates can be generated. Additionally, the virtual structures are passed to a physics-based simulation tool in order to generate virtual scanning transmission electron microscopy (STEM) images. The preset parameters of the 3D model together with the simulated STEM images serve as a database for the training of convolutional neural networks, which can be used to determine the parameters of the underlying 3D model and, consequently, to predict 3D structures of hetero-aggregates from 2D STEM images. Furthermore, an error analysis is performed with respect to structural descriptors, e.g. the hetero-coordination number. The proposed method is applied to image data of TiO2-WO3 hetero-aggregates, which are highly relevant in photocatalysis processes. However, the proposed method can be transferred to other types of aggregates and to different 2D microscopy techniques. Consequently, the method is relevant for industrial or laboratory setups in which product quality is to be quantified by means of inexpensive 2D image acquisition.

Prediction of sloshing pressure using image-based deep learning

This study investigates an image-based deep learning method for predicting various types of sloshing pressure, including regular, weak impulse, random, and peak pressures. The time series of pressure at a specific location on a tank wall are estimated from independent features of the sequential images of free-surface waves in sloshing flow. A dataset of wave images labeled with pressure is inserted into the residual neural network (ResNet) to enable the training of neural networks without encountering the problem of vanishing gradients. The hyperparameters of the ResNet algorithm are tuned to determine the optimal learning rate, mini-batch size, and strength of L1 regularization. Compared with experimental measurements, the prediction performance regarding the time history of pressure is remarkably good in the non-resonance regime, and the predictions exhibit reasonable agreement in the resonance regime. Notably, the method for pressure normalization has a significant impact on the prediction performance for peak pressure in the resonance regime, affecting both the magnitude and profile of the pressure. Furthermore, to enable practical applications of the proposed model, training is conducted using a comprehensive dataset of wave images including all pressure types. The model successfully estimates the time history for the different pressure types that occur sequentially.

A New Approach to Structural Damage Identification Based on Power Spectral Density and Convolutional Neural Network

In the field of structural health monitoring, vibration-based damage identification remains a formidable challenge. Key to this challenge is the establishment of a reliable association between observed vibration characteristics and the actual state of structural damage (e.g. stiffness reduction). This association not only accurately indicates the presence of damage, but also the location and severity of the damage. To solve this complex pattern identification problem, a large number of approaches, including deep learning, have emerged in recent years. In this paper, we propose a new structural damage identification method that utilizes the vibration information of the structure and a convolutional neural network based on Alex NET improvement. The method consists of calculating the acceleration response power spectral density of damaged and undamaged structures under impact loading separately, and then making a difference between the two power spectral data, and subsequently introducing these power spectral difference data into the convolutional neural network for training. The use of power spectral density analysis as a preprocessing step converts the time-domain signals into frequency-domain signals, and this conversion allows the convolutional neural network to capture and learn from the specific frequency characteristics of the data, thus facilitating the learning process of the neural network model. In this paper, the effectiveness of the method is critically evaluated through numerical simulation and experimental validation, and 3% and 5% noise are added to the numerical study to test the robustness of the method. During the convolution neural network training process, the optimal training mean squared error (MSE) is [Formula: see text] in the case of no noise addition; the optimal training MSE is [Formula: see text] in the case of noise addition. Both the results of simulations and experiments confirm the high accuracy and good robustness of the method in localizing structural damage.

Principal Component Networks: Utilizing Low-Rank Activation Structure to Reduce Parameters Early in Training

Recent works show that overparameterized neural networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight the potential to reduce the computational costs of deep neural network training without sacrificing generalization performance. Initial approaches for finding these small networks relied on expensive multi-round train-and-prune procedures, limiting their practical potential, but more recent work identifies subnetworks using structured pruning techniques early in training. In this article, we study network activations, rather than network weights, and find that hidden layer activations in overparameterized networks exist primarily in subspaces smaller than the actual model width. We further notice that these subspaces can be identified early in training. Based on these observations, we show how to efficiently find small networks that exhibit similar accuracy to their overparameterized counterparts after only a few training epochs. We term these network architectures Principal Component Networks (PCNs). PCNs compress individual layers by retaining only the high variance linear combinations of channels—defined by the principal components of the layer inputs—a key difference from structured pruning techniques which focus on individual channel pruning using localized channel measurements. We evaluate PCNs on CIFAR-10 and ImageNet for VGG and ResNet style architectures and compare against existing methods for subnetwork identification during early training. We find that PCNs consistently reduce parameter counts with little accuracy loss, thus providing the potential to reduce the computational costs of deep neural network training. Beyond model compression, we also connect our observation regarding hidden layer activations to the feature representations learned by neural networks and discuss areas for future work.